Overlap Aware Reordering of Rendering Operations for Efficiency

ABSTRACT

Disclosed are apparatus and methods for rendering using a graphics processing component (GPC). A computing device can receive instructions for a GPC, including an instruction IA associated with a first portion of a canvas. An insertion position in an instruction buffer for instruction IA can be determined by: determining an instruction IB in the instruction buffer that is associated with a second portion of the canvas. If the first and second portions overlap, the insertion position can be based on an overlapping-instruction position of IB in the instruction buffer. Otherwise, if instructions IA and IB are similar, then the insertion position can be based on a second position of IB in the instruction buffer. Otherwise, the insertion position can be determined based on an ending position of the instruction buffer. Instruction IA can be inserted in the instruction buffer at the insertion position.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Patent Application No.61/886,643, filed Oct. 3, 2013, entitled “Overlap Aware Reordering ofRendering Operations for Efficiency”, the contents of which are fullyincorporated by reference herein for all purposes.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Many modern computing devices, including mobile phones, personalcomputers, and tablets, provide graphical user interfaces (GUIs) forpermitting users to interact with the computing device. For example,application programs can use the GUI to communicate with a user usingimages, text, and graphical elements such as windows, dialogs, pop-ups,images, buttons, scrollbars, and icons. The GUI can also receive inputsfrom user-interface devices such as touch screens, computer mice,keyboards, and other user-interface devices to permit the user tocontrol the GUI, and thus the application program.

In some cases, the GUI can be used to interact with an operating system(OS) to manage the computing device. For example, the OS can have acontrol panel or setting application that uses the GUI to draw one ormore windows related to control settings for some aspect(s) of thecomputing device, such as audio controls, video outputs, computermemory, and human language(s) used by the OS (e.g., choose to receiveinformation in English, French, Mandarin, Hindi, Russian, etc.). Thecontrol panel/settings application can receive subsequent input relatedto the window(s) using the GUI. The GUI can provide the inputs to theOS, via the control panel/settings application, to manage the computingdevice.

SUMMARY

In one aspect, a method is provided. A computing device receives one ormore instructions for a graphics processing component of the computingdevice. Each instruction is associated with at least a portion of acanvas. After receipt of each instruction of the one or moreinstructions, the computing device determines a first portion of thecanvas associated with the instruction. The computing device determinesan insertion position in an instruction buffer for the instruction isdetermined by at least: determining an instruction in the instructionbuffer and a second portion of the canvas associated with theinstruction in the instruction buffer; if the second portion overlapsthe first portion, then the computing device determines the insertionposition based on an overlapping-instruction position, where theoverlapping-instruction position is based on a position in theinstruction buffer of the instruction in the instruction buffer;otherwise if the instruction in the instruction buffer is identified asbeing similar to the instruction, then the computing device determinesthe insertion position based on a second position, where the secondposition is based on the position in the instruction buffer of theinstruction in the instruction buffer; otherwise the computing devicedetermines the insertion position based on an ending position of theinstruction buffer. The instruction in the instruction buffer isinserted at the insertion position. For each instruction in theinstruction buffer, the computing device provides the instruction to thegraphics processing component, and the graphics processing componentperforms the instruction.

In another aspect, a computing device is provided. The computing deviceincludes a graphics processing component, a processor, and anon-transitory computer-readable storage medium having instructionsstored thereon that, when executed by the processor, cause the computingdevice to perform functions. The functions include: (A) receiving one ormore instructions for the graphics processing component, where eachinstruction is associated with at least a portion of a canvas, (B) afterreceipt of each instruction of the one or more instructions: (B1)determining a first portion of the canvas associated with theinstruction; (B2) determining an insertion position in the instructionbuffer for the instruction by at least: (B2 a) determining aninstruction in the instruction buffer; (B2 b) determining a secondportion of the canvas associated with the instruction in an instructionbuffer; (B2 c) if the second portion overlaps the first portion, thendetermining the insertion position based on an overlapping-instructionposition, where the overlapping-instruction position is based on aposition in the instruction buffer of the instruction in the instructionbuffer; (B2 d) otherwise if the instruction in the instruction buffer isidentified as being similar to the instruction, then determining theinsertion position based on a second position, where the second positionis based on the position in the instruction buffer of the instruction inthe instruction buffer; (B2 e) otherwise, determining the insertionposition based on an ending position of the instruction buffer; and (C)inserting the instruction in the instruction buffer at the insertionposition.

In another aspect, an article of manufacture is provided. The article ofmanufacture includes a non-transitory computer-readable storage mediumhaving instructions stored thereon that, when executed by a processor ofa computing device, cause the computing device to perform functions. Thefunctions include: (A) receiving one or more instructions for thegraphics processing component, where each instruction is associated withat least a portion of a canvas, (B) after receipt of each instruction ofthe one or more instructions: (B1) determining a first portion of thecanvas associated with the instruction; (B2) determining an insertionposition in the instruction buffer for the instruction by at least: (B2a) determining an instruction in the instruction buffer; (B2 b)determining a second portion of the canvas associated with theinstruction in an instruction buffer; (B2 c) if the second portionoverlaps the first portion, then determining the insertion positionbased on an overlapping-instruction position, where theoverlapping-instruction position is based on a position in theinstruction buffer of the instruction in the instruction buffer; (B2 d)otherwise if the instruction in the instruction buffer is identified asbeing similar to the instruction, then determining the insertionposition based on a second position, where the second position is basedon the position in the instruction buffer of the instruction in theinstruction buffer; (B2 e) otherwise, determining the insertion positionbased on an ending position of the instruction buffer; and (C) insertingthe instruction in the instruction buffer at the insertion position.

In another aspect, a device is provided. The device includes: means forreceiving one or more instructions for a graphics processing component,where each instruction is associated with at least a portion of a canvasand means for, after receipt of each instruction of the one or moreinstructions: determining a first portion of the canvas associated withthe instruction; receiving one or more instructions for the graphicsprocessing component, where each instruction is associated with at leasta portion of a canvas, after receipt of each instruction of the one ormore instructions: determining a first portion of the canvas associatedwith the instruction; determining an insertion position in theinstruction buffer for the instruction by at least: determining aninstruction in the instruction buffer; determining a second portion ofthe canvas associated with the instruction in an instruction buffer; ifthe second portion overlaps the first portion, then determining theinsertion position based on an overlapping-instruction position, wherethe overlapping-instruction position is based on a position in theinstruction buffer of the instruction in the instruction buffer;otherwise if the instruction in the instruction buffer is identified asbeing similar to the instruction, then determining determines theinsertion position based on a second position, where the second positionis based on the position in the instruction buffer of the instruction inthe instruction buffer; otherwise, determining the insertion positionbased on an ending position of the instruction buffer; and means forinserting the instruction in the instruction buffer at the insertionposition.

In one aspect, a method is provided. A computing device receives one ormore instructions for a graphics processing component of the computingdevice. Each instruction is associated with at least a portion of acanvas. After receipt of each instruction of the one or moreinstructions, the computing device determines a first portion of thecanvas associated with the instruction. The computing device searches aninstruction buffer to identify a mergeable instruction for theinstruction. The computing device, after identifying the mergeableinstruction at a mergeable position within the instruction buffer,searches one or more instructions in the instruction buffer for overlapin the canvas with the first portion, where the one or more instructionsinclude the mergeable instruction. In the event that the one or moreinstructions in the instruction buffer do not overlap the first portionof the canvas, the computing device merges the instruction with themergeable instruction. For each instruction in the instruction buffer,the computing device provides the instruction to the graphics processingcomponent, and the graphics processing component performs theinstruction.

In another aspect, a computing device is provided. The computing deviceincludes a graphics processing component, a processor, and anon-transitory computer-readable storage medium having instructionsstored thereon that, when executed by the processor, cause the computingdevice to perform functions. The functions include: (A) receiving one ormore instructions for the graphics processing component, where eachinstruction is associated with at least a portion of a canvas; and (B)after receipt of each instruction of the one or more instructions: (B1)determining a first portion of the canvas associated with theinstruction; (B2) searching an instruction buffer to identify amergeable instruction for the instruction; and (B3) after identifyingthe mergeable instruction at a mergeable position within the instructionbuffer: (B3 a) searching one or more instructions in the instructionbuffer for overlap in the canvas with the first portion, where the oneor more instructions comprise the mergeable instruction, and (B3 b) inthe event that the one or more instructions in the instruction buffer donot overlap the first portion of the canvas, merging the instructionwith the mergeable instruction.

In another aspect, an article of manufacture is provided. The article ofmanufacture includes a non-transitory computer-readable storage mediumhaving instructions stored thereon that, when executed by a processor ofa computing device, cause the computing device to perform functions. Thefunctions include: (A) receiving one or more instructions for a graphicsprocessing component of the computing device, where each instruction isassociated with at least a portion of a canvas; and (B) after receipt ofeach instruction of the one or more instructions: (B1) determining afirst portion of the canvas associated with the instruction; (B2)searching an instruction buffer to identify a mergeable instruction forthe instruction; and (B3) after identifying the mergeable instruction ata mergeable position within the instruction buffer: (B3 a) searching oneor more instructions in the instruction buffer for overlap in the canvaswith the first portion, where the one or more instructions comprise themergeable instruction, and (B3 b) in the event that the one or moreinstructions in the instruction buffer do not overlap the first portionof the canvas, merging the instruction with the mergeable instruction.

In another aspect, a device is provided. The device includes: means forreceiving one or more instructions for means for graphics processing,where each instruction is associated with at least a portion of acanvas; and means for, after receipt of each instruction of the one ormore instructions: determining a first portion of the canvas associatedwith the instruction; searching an instruction buffer to identify amergeable instruction for the instruction; and after identifying themergeable instruction at a mergeable position within the instructionbuffer: searching one or more instructions in the instruction buffer foroverlap in the canvas with the first portion, where the one or moreinstructions comprise the mergeable instruction, and in the event thatthe one or more instructions in the instruction buffer do not overlapthe first portion of the canvas, merging the instruction with themergeable instruction.

BRIEF DESCRIPTION OF THE FIGURES

In the figures:

FIG. 1A is a flow chart of a method, in accordance with an exampleembodiment.

FIG. 1B is a flow chart of another method, in accordance with an exampleembodiment.

FIGS. 2A and 2B are block diagrams of an example mobile device with agraphics processing component, in accordance with an example embodiment.

FIG. 3 shows a user interface scenario for an example messagingapplication, in accordance with an example embodiment.

FIG. 4A shows another user interface scenario for the example messagingapplication, in accordance with an example embodiment.

FIG. 4B shows yet another user interface scenario for the examplemessaging application, in accordance with an example embodiment.

FIG. 5 shows a user interface scenario for an example drawingapplication, in accordance with an example embodiment.

FIG. 6 depicts a distributed computing architecture, in accordance withan example embodiment.

FIG. 7A is a block diagram of a computing device, in accordance with anexample embodiment.

FIG. 7B depicts a cloud-based server system, in accordance with anexample embodiment.

DETAILED DESCRIPTION

Overview

Disclosed herein are techniques for reordering graphics processinginstructions for efficiency. Many computing devices, such as many modernmobile phones, are equipped with one or more graphics processingcomponents, such as one or more Graphics Processing Units (GPUs), andrelated software for utilizing the graphics processing components, suchas graphics libraries; e.g., OpenGL®.

Many graphics processing units are configured to receive and executeinstructions to draw images on a canvas. Examples of these instructionsinclude instructions to draw text, draw an image from an image file orbitmap, render a list of items, translate (i.e., move) a point of view,and specify a clipping (i.e., visibility) rectangle. A canvas can be anarea of memory and/or other components configured for storing, drawing,and/or displaying graphical data. Examples of the canvas include, butare not limited to, displays, frame buffers, video buffers, drawables,drawable regions, and video memory.

The graphics processing component can use a “rendering pipeline” to drawtext, shapes, images, etc. to the canvas. The rendering pipeline uses acollection of components to execute the instructions for the graphicsprocessing component to generate an image stored in a frame bufferacting as a canvas. The frame buffer can store pixel data for each pixelof a display, where one pixel, or picture element, is the smallestchangeable portion of a display. Each pixel in the frame buffer can bemapped to a corresponding pixel of the display, so that changing datafor a pixel in the frame buffer can cause the display to correspondinglychange the light emitted to display the pixel. For example, consider adisplay of size R rows and C columns, or in other words, a display thathas R×C pixels. A corresponding frame buffer that is mapped to thedisplay can include storage for each of the R×C pixels.

The rendering pipeline can include “shaders”, which are programsprovided by graphics libraries and/or applications using the graphicspipeline. Each shader allows the application to customize part of thegraphics pipeline by controlling processing of data in the graphicspipeline. For example, a shape can be specified to the graphics pipelinebased on a number of points that can be referred to as vertices. As anexample, a vertex shader can be used to control the color of a vertex.To draw a blue rectangle specified using four vertices (one per corner),the vertex shader can specify that each vertex of the rectangle shouldbe colored blue. Other components of the graphics pipeline arepredetermined components, such as a rasterizer that can convert aprimitive into a patch, or collection of pixels, that represent theprimitive.

The order of instructions provided to the graphics processing componentcan affect rendering performance. For example, a first set of shaderscan be associated with a first instruction and a second set of shaderscan be associated with a second instruction. The graphics processingunit can perform the first instruction using the first set of shaders.To perform the second instruction, the graphics processing unit canswitch out the first set of shaders and switch in the second set ofshaders. However, switching shaders consumes time and graphicsprocessing component resources, which affects rendering performance. Asanother example, other aspects of graphics processing that change on aper-instruction basis can affect rendering performance. Examples ofthese aspects include, but are not limited to, shader input (e.g., amatrix, a color), a bitmap/texture being used, a clipping region, and ablending mode.

Similar instructions tend to use the same shaders and have the sameaspects of graphics processing. Thus, similar instructions can be sortedand/or merged to reduce the amount of time and resources used forgraphics processing. The similarity of instructions can be determinedbased on a rendering state of the graphics pipeline for each operation.Sorting and/or merging instructions based on similarity can reduce theoverhead; e.g., time and resources used in switching shaders to draw animage.

One technique for determining similarity of instructions is based on avisible object type (VOT) assigned to each instruction. Instructions canbe sorted by the associated VOT. VOTs can generally correspond torendering states, shader selection, and use of similar graphicsprimitives. For example, suppose a rendering pipeline supports DrawRect,DrawTriangle, and DrawText instructions, where the DrawRect instructionrenders one or more rectangles, the DrawTriangle instruction renders oneor more triangles, and the DrawText instruction draws text. The DrawRectand DrawTriangle instructions can each use one set of graphicsprimitives and shaders to draw triangles and rectangles, while theDrawText instruction can use a different set of graphics primitives andshaders to draw text. As rectangles and triangles are both shapes, a VOTof “shape” can be used for both the DrawRect and DrawTriangleinstructions, while a VOT of “text” can be used for the DrawTextinstruction Then, if the same shaders and graphics primitives are usedby all instructions having a common VOT, sorting instructions by VOT cangenerate a group of instructions that uses the same shaders. Then, theshaders and primitives need only be changed, at most, just before andjust after the group of instructions having the same VOT.

In some embodiments, the instructions can be combined to performmultiple operations with a single instruction. For example, a singledraw text instruction can be provided with multiple texts andcorresponding display locations to render all of texts on a display. Bycombining instructions, any per-instruction overhead can be reduced asthe number of instructions is reduced.

Overlap of instructions can be considered to ensure an image drawn usingmerged and/or sorted instructions is correct. For example, suppose aseries of instructions for a graphics processing component are orderedfor first-to-last drawing—that is, the first received instruction is tobe drawn first, then the second received instruction is drawn at thesame time as or after the first instructions, and so on until the lastreceived instruction is drawn. Changing an order of drawing instructionsordered for first-to-last drawing can cause an image to be incorrectlydrawn.

To preserve first-to-last ordering and to take overlap into account,received instructions can be stored in an instruction buffer. Uponreceipt of a new instruction, the instruction buffer can be searched inreverse order for sorting and/or merging the new instruction withinstruction(s) in the instruction buffer. This search can takeoverlapping drawing regions of the canvas associated with the newinstruction into account. By searching in reverse order and takingoverlapping into account, first-to-last drawing ordering can bepreserved, and so resulting images can be drawn correctly.

Further, combinations of sorting and merging instructions canadditionally reduce overhead. For example, when a plurality ofoperations are sorted and then merged, the sorting step can bringsimilar instruction together into a consecutive group of operations.Then, subsequently merging the consecutive group of instructions canreplace the consecutive group of instructions with potentially only oneinstruction. This combination of sorting and then merging instructionscan reduce overhead both due to changing shaders and in processinginstructions.

Example Operations

FIG. 1A is a flow chart of method 100, in accordance with an exampleembodiment. Method 100 begins at block 110, where a computing device canreceive one or more instructions for a graphics processing component ofthe computing device, where each instruction is associated with at leasta portion of a canvas.

At block 120, the computing device can begin a loop to process eachinstruction in the received one or more instructions. If no instructionsremain to be processed, the loop can terminate and method 100 canproceed to block 132. In some examples, instructions can arrive one at atime, and so the loop need only iterate once to process the oneinstruction.

At block 122, the computing device can determine a first portion of thecanvas associated with the instruction.

At block 124, the computing device can determine an insertion positionfor the instruction in an instruction buffer by carrying out at leastthe techniques of blocks 126, 128, and 130.

At block 126, the computing device can determine an instruction in theinstruction buffer and can determine a second portion of the canvasassociated with the instruction.

At block 128, if second portion overlaps first portion, then thecomputing device can determine the insertion position based on anoverlapping-instruction position. The overlapping-instruction positioncan be based on a position in the instruction buffer of the instructionin the instruction buffer. Otherwise, if the instruction in theinstruction buffer is identified as being similar to the instruction,then the computing device can determine the insertion position based ona second position. The second position can be based on the position inthe instruction buffer of the instruction in the instruction buffer.Otherwise, the computing device can determine the insertion positionbased on an ending position of the instruction buffer.

In some embodiments, the instruction can be associated with a VOT. Inthese embodiments, determining whether the instruction in theinstruction buffer is identified as being similar to the instruction caninclude: determining a VOT associated with the instruction in theinstruction buffer; determining whether the VOT associated with theinstruction in the instruction buffer is the particular VOT; and afterdetermining that the VOT associated with the instruction in theinstruction buffer is the particular VOT, determining that theinstruction in the instruction buffer is similar to the instruction. Inparticular of these embodiments, the VOT can be a type selected from thegroup consisting of a point type, a line type, a shape type, an imagetype, and a text type.

In other embodiments, the instruction has a beginning position thatbegins the instruction buffer, the ending position can end theinstruction buffer, and the instruction buffer can store at least oneinstruction. Then, determining the insertion position can include:setting a search position to an initial search position that is based onthe ending position; determining an instruction at the search positionin the instruction buffer; determining a third portion of the canvasassociated with the instruction at the search position of theinstruction buffer; determining whether the third portion overlaps thefirst portion; after determining that the third portion overlaps thefirst portion, determining that the overlapping-instruction position isthe search position; and otherwise: determining whether the instructionat the search position of the instruction buffer is similar to theinstruction, and after determining that the instruction at the searchposition of the instruction buffer is identified as being similar to theinstruction, then determining that the second position is the searchposition.

In particular of the other embodiments, method 100 can further include:after determining that the third portion does not overlap the firstportion and determining that the instruction at the search position ofthe instruction buffer is not similar to the instruction: determiningwhether the instruction at the search position of the instruction bufferis a most-similar instruction, where the most-similar instruction is aninstruction that is closest in similarity to the instruction and thathas been searched in the instruction buffer; if the instruction at thesearch position of the instruction buffer is a most-similar instruction,then storing a position in the instruction buffer associated with themost-similar instruction; and decrementing the search position.

In some of the particular of the other embodiments, after decrementingthe search position, determining whether the search position is beforethe beginning position; and after determining that the search positionis before the beginning position, inserting the instruction into theinstruction buffer at the position in the insertion buffer associatedwith the most-similar instruction.

In other embodiments, determining the insertion position for theinstruction based on the second position includes determining theinsertion position to be a position immediately after the secondposition.

At block 130, the computing device can insert the instruction in theinstruction buffer at the insertion position. After completing block130, method 100 can proceed to block 120.

At block 132, for each instruction in the instruction buffer, thecomputing device can provide the instruction to the graphics processingcomponent. The graphics processing component then can perform theinstruction. In some embodiments, the graphics processing component canperform the instruction using a rendering pipeline, where the renderingpipeline includes one or more shaders.

In particular of these embodiments, providing the instruction to thegraphics processing component can include providing information about ashader for use in the rendering pipeline. In some of the particularembodiments, providing information about the shader for use in therendering pipeline can include providing the shader to the graphicsprocessing component. In other particular of these embodiments,providing information about the shader for use in the rendering pipelineincludes providing, to the graphics processing component, a reference tothe shader. In even other of these embodiments, the one or more shadersinclude a shader selected from the list of shaders comprising a vertexshader, a geometry shader, and a fragment shader. In still otherembodiments, providing information about the shader can includedetermining the shader based on the VOT associated with the instruction.

Method 100 can end when all of the instructions in the instructionbuffer are performed.

FIG. 1B is a flow chart of method 150, in accordance with an exampleembodiment. Method 150 begins at block 160, where a computing device canreceive one or more instructions for a graphics processing component ofthe computing device, where each instruction is associated with at leasta portion of a canvas.

At block 170, the computing device can begin a loop to process eachinstruction in the one or more instructions. If no instructions remainto be processed, the loop can terminate and method 150 can proceed toblock 180.

At block 172, the computing device can determine a first portion of thecanvas associated with the instruction.

At block 174, the computing device can search an instruction buffer toidentify a mergeable instruction for the instruction.

At block 176, after identifying the mergeable instruction at a mergeableposition within the instruction buffer, the computing device can: searchone or more instructions in the instruction buffer for overlap in thecanvas with the first portion, where the one or more instructionscomprise the mergeable instruction. In the event that the one or moreinstructions in the instruction buffer do not overlap the first portionof the canvas, the computing device can merge the instruction with themergeable instruction.

After completing block 176, method 150 can proceed to block 170.

In some embodiments, the instruction buffer has a beginning positionthat begins the instruction buffer and an ending position that ends theinstruction buffer, and the instruction buffer can store at least oneinstruction. In these embodiments, searching the one or moreinstructions in the instruction buffer for overlap in the canvas withthe first portion can include: setting an overlap-search position to theending position of the instruction buffer. Then, at the overlap-searchposition: an overlap-search instruction at the overlap-search positioncan be determined; a second portion of the canvas associated with theoverlap-search instruction can be determined; in the event that thefirst portion of the canvas overlaps the second portion, a determinationcan be made that the searched one or more instructions do overlap theinstruction; otherwise: the overlap-search position can be decrementedand, in the event that the decremented overlap-search position is lessthan the mergeable position, a determination can be made that thesearched one or more instructions do not overlap the instruction.

In particular embodiments, the instruction can be associated with aparticular VOT. Then, searching an instruction buffer to identify themergeable instruction for the instruction can include: forming a mergingkey for the instruction, wherein the merging key comprises the VOT forthe instruction; performing a hash lookup in a hash table associatedwith the instruction buffer utilizing the merging key; and in responseto finding a match in the hash table for the merging key, determiningthat the mergeable instruction is an instruction in the hash tableassociated with the merging key.

In specific of these embodiments, the VOT can be a type selected fromthe group consisting of a point type, a line type, a shape type, animage type, and a text type.

In still other embodiments, merging the instruction and the secondinstruction can include: the computing device generating a mergedinstruction by merging the instruction and the mergeable instruction;and the computing device replacing the mergeable instruction in theinstruction buffer with the merged instruction.

At block 180, for each instruction in the instruction buffer, thecomputing device can provide the instruction to the graphics processingcomponent. The graphics processing component then can perform theinstruction.

In some embodiments, the graphics processing component performing theinstruction can include the graphics processing component performing theinstruction using a rendering pipeline, where the rendering pipelineincludes one or more shaders. In particular of these embodiments,providing the instruction to the graphics processing component caninclude providing information about a shader for use in the renderingpipeline. In some of the particular embodiments, providing informationabout the shader for use in the rendering pipeline can include providingthe shader to the graphics processing component. In other particular ofthese embodiments, providing information about the shader for use in therendering pipeline includes providing, to the graphics processingcomponent, a reference to the shader. In even other of theseembodiments, the one or more shaders include a shader selected from thelist of shaders comprising a vertex shader, a geometry shader, and afragment shader. In still other embodiments, providing information aboutthe shader can include determining the shader based on the VOTassociated with the instruction.

In particular of these embodiments, method 150 additionally can include:after determining that the searched one or more instructions do overlapthe instruction: determining a most-similar instruction of the searchedone or more instructions, wherein the most-similar instruction is aninstruction that is closest in similarity to the instruction;determining a most-similar position in the instruction buffer for themost-similar instruction; and inserting the instruction into theinstruction buffer at a position that is based on the most-similarposition.

Method 150 can end when all of the instructions in the instructionbuffer are performed.

Example Graphics Processing Components

FIGS. 2A and 2B are block diagrams of example mobile device 210 with agraphics processing component 720, in accordance with an exampleembodiment. FIG. 2A is a block diagram of mobile device 210 with agraphics processing component 720 configured to provide output to acanvas 230.

Mobile device 210 can be a computing device, such as computing device700, configured for graphics processing. For example, mobile device 210can include graphics processing component (GPC) instruction buffer 220for storing instructions for graphics processing component 720. Eachinstruction can be associated with a VOT, such as, but not limited to,type(s) for a clip rectangle, type(s) for point(s) of view, type(s) fordrawable or visible object(s), type(s) for lists of drawable objects,and combinations thereof.

Drawable or visible objects can include, but are not limited, to points,lines, shapes, images, and text. Instruction(s) related to a point ofview, such as a point of view for viewing/displaying drawable object(s),can include, but are not limited to, instruction(s) to translate, ormove, the point of view, instruction(s) to rotate the point of view, andcombinations thereof. Instruction(s) associated with lists of objectscan include instruction(s) for managing and drawing lists of objects,instruction(s) to create a list of drawable objects, instruction(s) toadd items to the list of drawable objects, instruction(s) to removeitems from the list of drawable objects, instruction(s) to update one ormore items in the list of drawable objects, instruction(s) to delete alist of drawable objects, instruction(s) to draw some or all of theobjects in the list of drawable objects, and combinations thereof.

Instruction(s) associated with drawable object(s) can include, but arenot limited to, instruction(s) to draw one or more points,instruction(s) to draw one or more lines, instruction(s) to draw one ormore shapes, instruction(s) to draw one or more images, instruction(s)to draw one or more text items, and combinations thereof. Instruction(s)associated with images can include instruction(s) to draw a bitmap, orrepresentation of pixels in an image where one or more bits in thebitmap correspond to one pixel in the image. Instruction(s) fortransforming objects can include, but are not limited to, instruction(s)to rotate one or more drawable objects, instruction(s) to translate oneor more drawable objects, instruction(s) to scale one or more drawableobjects, and combinations thereof.

Mobile device 210 can receive graphics processing componentinstructions, store the instruction in graphics processing componentinstruction buffer 220, and operate on the stored instructions ingraphics processing component instruction buffer 220.

FIG. 2A shows that mobile device 210 can sort instructions for agraphics processing component by similarity. The similarity ofinstructions can be determined based on a rendering state of thegraphics pipeline for each operation. The rendering state can includeaspects of the graphics pipeline, including but not limited to:shader(s) used by the instruction, shader parameter(s) used by theinstruction, transformation matrix/matrices for the instruction, a pointof view for the instruction, partial or complete shape(s) drawn by theinstruction, a type of the instruction, a portion of the canvasassociated with the instruction, and combinations thereof.

For one example of similarity, a bounded image can be drawn by drawing aboundary rectangle (or other shape) and mapping image data from one ormore texture maps, or associated image(s), to pixel(s) in an interior ofthe boundary rectangle. In this case, instructions associated withtexture maps can be considered to be similar.

Another technique for determining similarity of instructions is based ona visible object type, or VOT, assigned to each instruction. VOTs cangenerally correspond to rendering states. For example, if the sameshaders are used by all instructions sharing a common VOT, sortinginstructions by VOT can generate a group of instructions all having thecommon VOT, and thus all of the instructions in the group can use thesame shaders. Then, the shaders need only be changed, at most, justbefore and just after the group of instructions having the same VOT.

Sorting instructions by similarity can take rendering state and perhapscanvas overlap into account. For example, suppose that theinstruction(s) for drawing drawable objects include a “DrawPoint”instruction for drawing points with an example VOT of “Point”, a“DrawLine” instruction for drawing lines with an example VOT of “Line”,a “DrawShape” instruction for drawing shapes with an example VOT of“Shape”, a “DrawBitmap” instruction to draw images stored as bitmapswith an example VOT of “Image”, and a “DrawText” instruction to drawtext with an example VOT of “Text”. Then, mobile device 210 can sortinstructions stored in graphics processing component instruction buffer220 so that instructions with the same VOT are next to each other ingraphics processing component instruction buffer 220, the DrawBitmapinstructions, each having a VOT of “Image”, are next to each other ingraphics processing component instruction buffer 220, and so on.

A series of instructions for a graphics processing component can beordered for first-to-last drawing. For example, suppose a series ofinstructions for a graphics processing component I1, I2, . . . , In,n>2, are provided to a computing device, where each instruction isassociated with at least a portion of a canvas. If the instructions I1,I2, . . . , In are ordered for first-to-last drawing, then (i)instruction I1 is drawn first, since I1 is received first, (ii)instruction I2 is drawn at the same time as or after instruction I1,since I2 is received after I1, (iii) instruction I3 is drawn at the sametime as or after instruction I2, since I3 is received after I1 and I2,and so on until (n) the last instruction In is drawn.

In some cases, changing an order of drawing instructions ordered forfirst-to-last drawing can cause an image to be incorrectly drawn. Forexample, suppose the image includes two shapes: S1 and S2, with S2 drawnpartially on top of S1, and the instructions include a first instructionto DrawShape(S1) and a second instruction to DrawShape(S2). S1 and S2can be said to overlap, since S2 at least partially covers S1. Changingthe order of the DrawShape instructions; e.g., performing theDrawShape(S2) instruction before the DrawShape(S1) instruction, cancause the resultant image to be incorrect, since S2 would be drawn belowS1 rather than correctly being drawn partially on top of S1. Correctnessof an image drawn using instructions involving overlapping drawableobjects can be preserved when the order of drawing the overlappingdrawable objects is preserved. In this example, correctness of the imageis preserved if S1 is drawn before S2.

To preserve first-to-last ordering and to take overlap into account,received instructions can be stored in an instruction buffer. Uponreceipt of a new instruction, the instruction buffer can be searched inreverse order for sorting and/or merging the new instruction withinstruction(s) in the instruction buffer taking overlapping drawingregions of the canvas for the instructions into account. If the newinstruction is not merged or inserted into the instruction buffer forsorting, the new instruction can be inserted at the end of theinstruction buffer or near a most-similar instruction. By searching inreverse order and taking overlapping into account, first-to-last drawingordering can be preserved, and so resulting images can be drawncorrectly.

In some embodiments, first-to-last drawing ordering can preserve depthordering (Z ordering) that is implied by the first-to-last drawingordering. In other embodiments, first-to-last drawing ordering may notpreserve depth ordering; for example, the instructions in theinstruction buffer can be used to draw a user interface or other imageryatop a frame of an already-rendered image where depth ordering can beset explicitly as part of the geometry, such as a frame of athree-dimensional video game or architectural/mechanical rendering.

In other cases, changing a state of the graphics processing componentwhile drawing an image can change the image. The state can be based oninformation, such as but not limited to, scaling information,translation information, shearing/skewing information, perspectiveinformation, and rotation information. The information about the stateof the image can be aggregated in a matrix. A matrix can be associatedwith each instruction of the state of the image.

For example, suppose three instructions are provided:

DrawBitmap B1 at 0, 0

TranslateY by 10 pixels

DrawBitmap B2 at 100, 0

Let M1 be a matrix associated with the first DrawBitmap command which isindicates that the first bitmap B1 is to be drawn at (x, y) position (0,0); i.e., without translation. The TranslateY command can indicate thatcoordinates for the image are to be moved, or translated, by 10 pixelsin the Y dimension—this translation information can be used to updatethe matrix M1 to a new matrix M2. Then, if M2 is used in drawing thesecond DrawBitmap command, the effect of drawing bitmap B2 at position(100, 0) includes a translation in the Y dimension of 10 pixels. Aftertranslation, bitmap B2 would be drawn at position (100, 10). In someembodiments, one matrix can be stored and updated rather than usingseparate matrices; e.g., a matrix M that is initialized to the values ofM1 and then updated with the above-mentioned translation in the Ydimension to have the values of M2.

In some embodiments, the matrix can be used to determine overlaps; e.g.,boundary areas associated with first and second instructions can betransformed by the matrix to determine respective first and secondregions of a canvas covered by the first and second instruction. Then,if the first and second regions overlap, the first and secondinstructions can be considered to be overlapping; otherwise, the firstand second instructions can be considered to be non-overlapping.

The order of instructions that change the state of the graphicsprocessing component can be preserved to ensure correctness of resultantimages. Also, the relative order of state change and drawinginstructions can be preserved to ensure correctness of resultant images.

One sorting technique for sorting instructions taking overlap and stateinto account can include mobile device 210 receiving an instruction NIfor graphics processing component 720. Mobile device 210 can maintain aninstruction buffer, such as graphics processing component instructionbuffer 220. To start the sorting technique, mobile device 210 canreceive the instruction NI. Mobile device 210 can then search to find aposition to insert NI into the instruction buffer. Mobile device 210 caninitially set a current position CP at an ending position EP storing aninstruction in the instruction buffer. For example, suppose that 3instructions I1, I2, and I3 are stored in the instruction buffer atrespective positions 1, 2, and 3. Then, the ending position EP storingan instruction is position 3, and so mobile device can start at position3. Also, a beginning position BP can be determined as a first positionstoring an instruction; e.g., the beginning position can be position 1.

The sorting technique can continue at a LOOPSTART location by letting CIbe a current instruction stored at position CP. If the currentinstruction CI changes a state of graphics processing component 720 orif an object drawn by instruction NI at least partially overlaps anobject drawn by instruction CI; i.e., instruction NI overlapsinstruction CI, then insert the new instruction NI at a positionimmediately after CP; e.g., at position CP+1.

One approach to approximately determine overlap between instructions NIand CI can involve determining a bounding box BNI that bounds the objectdrawn using instruction NI and determining a bounding box BCI thatbounds the object drawn using instruction CI. Then, it can be determinedthat instruction NI at least partially overlaps (or does not overlap)instruction CI when BNI at least partially overlaps (or does notoverlap) BCI. Other approaches for determining overlapping and notoverlapping instructions are possible as well. In some embodiments,multiple bounding boxes can be used, such as in a 3D rendering scenario.

The sorting technique can continue by comparing similarity criteria,such as but not limited to a VOT, associated with instruction NI withsimilarity criteria associated with instruction CI; e.g., the VOTassociated with an instruction can be stored in the instruction bufferalong with the instruction. If the similarity criteria associated withinstruction NI matches, or nearly matches, the similarity criteriaassociated with instruction CI, the new instruction NI can be insertedat a position immediately after CP. Otherwise, the similarity criteriaassociated with instruction NI does not match the similarity criteriaassociated with instruction CI—in this case, decrement the currentposition; e.g., let CP equal the value of CP−1. If CP is now at aposition before the beginning position BP, insert NI at beginningposition BP of the instruction buffer; e.g., if BP is 1 and CP is 0 (orless), then insert NI at BP=position 1 of the instruction buffer. If CPis not before BP, the sorting technique can loop back to theabove-mentioned LOOPSTART.

If no similarity criteria associated with instruction NI match (ornearly match) similarity criteria for any instruction stored in theinstruction buffer, then instruction NI can be stored at the endingposition EP. In other embodiments, when no similarity criteriaassociated with instruction NI match (or nearly match) similaritycriteria for any instruction stored in the instruction buffer, aninstruction MSI stored in the instruction buffer can be determined,where the similarity criteria for instruction MSI are the most similarto the similarity criteria associated with instruction NI. Then,instruction NI can be inserted into the instruction buffer a positionassociated with instruction MSI; e.g., directly after the position ofinstruction MSI.

For example, suppose that 3 instructions I1, I2, and I3 are stored inthe instruction buffer at respective positions 1, 2, and 3, each of I1,I2, and I3 having respective similarity criteria. Let MT be a matchingthreshold between similarity criteria of a new instruction, such as NI,and an instruction stored in the instruction buffer, where MT is, inthis example, set to 80% —other values of MT (e.g., 100%, 90%, 50%) arepossible as well. Thus, if the similarity criteria of instruction NImatch similarity criteria of an instruction stored in the instructionbuffer by at least MT %, then the instruction NI can be determined to(nearly) match the instruction stored in the buffer. In this example,suppose that similarity criteria for instruction NI and the similaritycriteria of each respective instruction I1, I2, I3 have match at 90%,25%, and 10%. As similarity criteria for instruction NI match thesimilarity criteria of instruction I1 at 90%, which exceeds thresholdMT, then NI can be determined to nearly match instruction I1. Then, NIcan be inserted into the instruction buffer at a position based on thematching instruction; e.g., directly after the position of instructionI1.

As another example, suppose that similarity criteria for instruction NIand the similarity criteria of each respective instruction I1, I2, I3have match at 20%, 25%, and 10%. Then, the similarity criteria forinstruction NI do not match the similarity criteria any storedinstruction by at least the threshold MT, then NI can be determined notto match, or not to be similar to any stored instruction. However, thehighest similarity criteria of 25% is between instruction NI andinstruction I2, so the most-similar instruction MSI in this example canbe instruction I2. Then, NI can be inserted into the instruction bufferat a position based on the position of the MSI; e.g., directly after theposition of instruction I2.

Once instructions for graphics processing component 720 have been sortedafter taking overlap and state into account as needed, instructions withthe same instruction can be merged, again while taking overlap and stateinto account. A merging technique can then be used to merge instructionsin an instruction buffer that takes overlap and state into account. Themerging technique can operate on an instruction buffer with beginningand ending positions as described above in the context of the sortingtechnique.

One merging technique can involve use of a hash table associated withthe instruction buffer. A merging key can be formed for each instructionreceived and then stored in the instruction buffer. In some embodiments,the merging key can be the VOT for the instruction. In otherembodiments, the merging key can include, but is not limited to: the VOTfor the instruction, a color associated with the instruction; e.g., atext color, a reference associated with the instruction; e.g., a pointerfor a bitmap, an anti-aliasing indicator associated with theinstruction, a blend-mode value/indicator, and combinations thereof.Other merging keys are possible as well.

After forming the merging key, the hash table can be searched bypresenting the merging key to the hash table. If no matching instructionis found in the hash table, then the instruction may not be mergeable,and so the instruction can be inserted into the instruction buffer;e.g., at the ending position EP of the instruction buffer. Otherwise, ifmatching instruction MI is found in the hash table, instruction MI canbe considered as a mergeable instruction. Let MI be stored at mergeableposition MP in the instruction buffer, where the beginning BP≦MP≦EP.Then, each instruction from mergeable position MP to ending position EPin the insertion buffer can be searched for overlap with the instructionusing the overlap determination techniques discussed above.

If no overlapping instruction is found between MP and EP, then theinstruction can be merged with the mergeable instruction MI at positionMP of the instruction buffer. If one of the instructions between MP andEP does overlap the instruction, then a most-similar instruction to theinstruction can be determined and the instruction can be inserted afterthe most-similar instruction. Similarity of instructions can bedetermined using the techniques discussed above. Another technique thatcan be used is to use a hash table with a simple merging key to theinstruction; e.g., VOT only, and to find a most-similar instruction MSIin the instruction buffer after the position MP that matches the simplemerging key. If instructions between a most-similar position MSP ofinstruction MSI and ending position EP do not overlap the instruction,then the instruction can be inserted after the MSP; otherwise theinstruction can be inserted at the EP.

Another merging technique can include initializing a value of a currentposition CP to the beginning position BP. The merging technique cancontinue at a LOOPSTART2 location, where the value of the currentposition CP can be compared to the ending position EP. If CP is greaterthan or equal to ending position EP, then the merging technique can end.Otherwise, let current instruction CI be the instruction at position CP,and let next instruction NI be the instruction at position CP+1. If aVOT associated with current instruction CI is the same as a VOTassociated with next instruction NI and if neither instruction CI norinstruction NI changes a state of graphics processing component 720 andif current instruction CI and next instruction NI do not overlap, thenmerge the next instruction NI into the current instruction CI.Otherwise, either the VOTs differ, the state is changed, or instructionsCI and NI overlap, so do not merge CI and NI; rather, increment thecurrent position CP; e.g., let a value of CP be set equal to the valueof CP+1. Then, after incrementing CP, the merging technique can loopback to the above-mentioned LOOPSTART2 location. Other sorting and/ormerging techniques are possible as well.

For example, suppose one instruction to graphics processing component720 is a DrawText instruction that takes a number of triples asparameters: an x value, a y value, and a text reference; e.g., a stringor pointer to string. Then, graphics processing component 720 canperform the DrawText instruction by drawing the “text reference” at ascreen position of “x value”, “y value”. Further suppose that, if two ormore triples are provided to the DrawText instruction; e.g.,DrawText(x1, y1, “text 1”, x2, y2, “text 2”, . . . ), then graphicsprocessing component 720 can draw each text reference in anindeterminate order e.g., sometimes “text 1” can be drawn before “text2” and other times “text 2” is drawn before “text 1”.

Continuing this example, suppose instructions in graphics processingcomponent instruction buffer 220 include three consecutive instructions:DrawText(0, 200, “text 1”); DrawText(0, 0, “text 2”); Draw Text(0, 100,“text 3”). In this example, let these three example text references eachtake 50 vertical or y pixels to draw, so that drawing text at (0, 200)does not overlap text drawn at (0, 0) or at (0, 100), since there ismore than 50 vertical pixels between y pixel position 200 and each of ypixel positions 0 and 100. However, text drawn at (0, 200) would overlaptext drawn at (0, 180) or (0, 220), since there are fewer than 50vertical pixels between y pixel position 200 and each of y pixelpositions 180 and 220.

Since the three consecutive instructions do not overlap, then the threeinstructions can be merged into a single DrawText instruction:DrawText(0, 200, “text 1”, 0, 0, “text 2”, 0, 100, “text 3”). However,if the DrawText(0, 0, “text 2”) instruction was replaced with aninstruction DrawText(0, 180, “text 2”), then DrawText(0, 200, “text 1”)could not be merged with the DrawText(0, 180, “text 2”) instructionsince the order of drawing the text references is indeterminate. In thisexample, the Draw Text(0, 100, “text 3”) instruction can still be mergedwith either the DrawText(0, 200, “text 1”) instruction or theDrawText(0, 180, “text 2”) instruction.

Sorting instructions by similarity and merging instructions can beperformed as indicated in FIG. 2A, which shows an example where sortingprecedes merging. In some embodiments, sorting can be performed withoutmerging; while in other embodiments, merging can be performed withoutsorting. In even other embodiments, merging can be performed beforesorting; e.g., to reduce the total number of instructions to be sorted.In particular of these even other embodiments, a second merging pass canbe performed after sorting to further reduce the number of instructionsafter sorting.

In other embodiments, sorting and merging can be performed as aspects ofone operation—for example, once two instructions have been determined tobe similar and non-overlapping, then the two similar and non-overlappinginstructions can be merged.

After the instructions in instruction buffer 220 have been sorted and/ormerged, the instructions in instruction buffer 220 can be provided tographics processing component 720 in an order from the beginningposition BP to ending position EP. In some embodiments, mobile device210 can provide a stream of instructions to graphics processingcomponent 720 for performance. In other embodiments, mobile device 210can provide a reference to instructions, such as a pointer or otherreference to graphics processing component instruction buffer 220, tographics processing component 720 for performance.

Upon receiving instruction(s) from mobile device 210, graphicsprocessing component 720 can perform the provided instructions to drawan image. For example as discussed immediately below in the context ofFIG. 2B, graphics processing component 720 can perform the instructionsusing a rendering pipeline to draw an image stored in a frame buffer, orregion of memory configured to store pixels of an image for display.

Canvas 230 includes an area of memory and/or other components configuredfor storing, drawing, and/or displaying graphical data. Examples ofcanvas 230 include, but are not limited to, displays, frame buffers,video buffers, drawables, drawable regions, and video memory. In someembodiments, providing graphical data for one or more portions of canvas230 can cause immediate or nearly immediate display. For example, ifcanvas 230 is or is associated with a display that displays 30 frames orimages per second, the provided graphical data can be displayed withinone (or, in particular scenarios, two) frame display intervals of 1/30thof a second.

Upon reception of a frame buffer, such as frame buffer 290, a displayassociated with canvas 230 can display the image stored in the framebuffer nearly immediately. In other embodiments, providing graphicaldata for one or more portions of canvas 230 does not cause the providedgraphical data to be (nearly) immediately displayed. For example, anexample canvas 230 can be an inactive frame buffer; i.e., currently notdisplayed frame buffer. When all graphical data for a desired image hasbeen provided to the inactive frame buffer, the inactive frame buffercan be made active; i.e., made available for display.

FIG. 2B is a block diagram of mobile device 210 with a graphicsprocessing component 720 that includes rendering pipeline 240. Renderingpipeline 240 includes vertex buffer 242, uniform attributes 246, vertexshader 250, geometry shader 260, clipping component 270, rasterizationcomponent 274, fragment shader 280, fragment processing component 284,and frame buffer 290. In some embodiments, frame buffer 290 is dedicatedto graphics pipeline 240, while in other embodiments, frame buffer 290is not dedicated to graphics pipeline 240; e.g., frame buffer 290 can bean area of memory provided by an application to graphics pipeline 240.

Vertex buffer 242 can include values of attributes for each of one ormore vertices, or points specifying object(s) to be rendered. Exampleobjects to be rendered include, but are not limited to, points, lines,shapes; e.g., triangles, groups of lines, groups of shapes, and images;e.g., textures. Example attributes of a vertex include a position of thevertex; e.g., in one dimensional (1D), two dimensional (2D), or threedimensional (3D) space, a color of the vertex, and a normal vector tothe vertex. Other attributes are possible as well. In some embodiments,mobile device 210 can provide vertices and vertex attributes related toinstructions for graphics processing component 720 using vertex buffer242. In these embodiments, graphics processing component 720 can operateon the vertices once mobile device 210 instructs graphics processingcomponent 720 to perform an instruction associated with the vertices andvertex attributes in vertex buffer 242.

Once an instruction involving vertices is started, vertices in vertexbuffer 242 are provided to vertex shader 250 as vertex attributes 244.Vertex shader 250 is software provided by an application using graphicsprocessing component 720 to customize its processing of vertices. Forexample, vertex shader 250 can determine a vertex's screen coordinatesin 2D. FIG. 2B shows that vertex shader 250 takes vertex attributes 244and uniform variables 246 as inputs and outputs vertex screencoordinates 252.

In some embodiments, the application can provide a shader to graphicsprocessing component 720 by providing information about the shader. Theinformation can be software for the shader; i.e., the application canprovide the shader directly to the graphics processing component 720, apointer or other reference to the shader; i.e., the application canprovide the shader indirectly to the graphics processing component 720,and/or a combination of software for and reference(s) to components ofthe shader.

Along with vertices from vertex buffer 242, vertex shader can accessuniform variables 246 during execution. Uniform variables are variablesset by the application using graphics processing component 720 tocommunicate with shaders, such as vertex shader 250. Vertex shader 250can read uniform variables to calculate the vertex's screencoordinates—for example, uniform variables specifying a point of viewcan be read by vertex shader 250 and used in calculating vertex screencoordinates 252. As another example, a transformation matrix thatspecifies an amount of object scaling; i.e., expansion or contraction,can be provided as a uniform variable to be used by vertex shader 250 incalculating vertex screen coordinates 252.

Many other example calculations by vertex shader 250 and other examplesof uniform variables 246 are possible as well.

In some embodiments, vertex screen coordinates 252 can have valueswithin predetermined limits. For example, visible vertex coordinates canbe provided with a ranges for a square having lower-left-cornercoordinates of (−1, −1) and upper-right-corner coordinates of (+1, +1),with vertices having coordinates outside of the square not beingvisible. Other example screen coordinates and limits are possible aswell.

Vertex screen coordinates 252 can be provided as an input to geometryshader 260 for generation of primitives 262. Primitives 262 arecollection of one or more vertices defining an object to be drawn by thegraphics processing component to perform the instruction. Exampleprimitives include, but are not limited to, point primitives, lineprimitives, groups of lines primitives, shapes primitives; e.g.,triangle primitives, and groups of shapes primitives. In someembodiments, geometry shader 260 can access uniform variables 246 tocalculate primitives 246.

Geometry shader 260 can generate zero or more primitives for inputvertex screen coordinates 252. Geometry shader 260 can provide multiplestreams of output; e.g., by generating a primitive once and outputtingthe generated primitive several times, such as once per output stream.For example, if drawing multiple images using the a set of recurringshapes; e.g., shapes in the background of a scene, primitives for therecurring shapes can be calculated once by geometry shader 260 and thenoutput to each of the multiple images.

In some embodiments not shown in the Figures, a tessellation stage ofgraphics pipeline 240 can be provided between vertex shader 250 andgeometry shader 260. Tessellation is a process of covering a shapewithout gaps or overlaps by simpler figures of one type (or a fewtypes). The tessellation stage can be used to take input vertices andgenerate primitives that tessellate a shape to be drawn by graphicsprocessing component 720. If provided, the tessellation stage can outputprimitives that are provided as inputs to geometry shader 260. As such,if no further processing is required, such primitives provided by thetessellation stage can be output by geometry shader 260 without change.

Once primitives 262 are generated, the primitives are compared to aclipping window, or region of visibility, during clipping processing270. A primitive within the clipping window is potentially visible inthe output image and so is output as a visible primitive of visibleprimitives 272. Primitives outside of the clipping window are discardedduring clipping processing 270.

Visible primitives 272 are then converted by rasterization processing274 to discrete image elements, such as fragments of pixels, for drawinginto frame buffer 290. Frame buffer 290 includes data for each pixel, orsmallest visible picture element that is ready for display. For example,in a color display having R rows and C columns of pixels, each usingRed/Green/Blue (RGB) to represent a color, frame buffer can have a redcolor value, a green color value, and a blue color value for each of theR*C pixels in the color display for a total of 3*R*C values: As anotherexample, the color display can also have a depth value associated witheach pixel, for a total for 4*R*C values in frame buffer 290. As such,drawing to frame buffer 290 is equivalent to drawing on the (color)display.

For a given visible primitive of visible primitives 272, rasterizationprocessing 274 takes vertex coordinates of vertices in the visibleprimitive and converts the vertex coordinates to display coordinates.For example, let vertex coordinates be specified within a (−1, −1) to(+1, +1) square as discussed above, and let the display have coordinates(0,0) in the lower left hand corner to (200, 200) in the upper righthand corner. Then, example functions to convert vertex coordinates (Xvc,Yvc) to display coordinates (x, y) are: x=(Xvc+1)*100 and y=(Yvc+1)*100.Then, each pixel in frame buffer 290 within the primitive is determined,and a color for the pixel determined. The color for the pixel can bedetermined by blending color and other values for each vertex in theprimitive. The pixels for the visible primitive can be output byrasterization processing 274 as a fragment of fragments 276.

Fragments 276 can be provided as input to fragment shader 280, which candetermine a final color for pixels in the fragment. For example,fragment shader 280 can apply a texture, or overlying image, to thefragment. As another example, fragment shader 280 can simulate how lightcan reflect off of a material making up the fragment to determine colorvalues for pixels in the fragment.

In some embodiments, graphics pipeline 240 can utilize and determinedepth values for pixels as part of determining which aspects of a givenshape are visible. For example, depth values can be determined on aper-pixel basis, where the higher the depth value associated with agiven color of a pixel, the “deeper” the pixel is and the less likelythe given color is to be visible. A fragment with pixel P having colorCP at a depth D can be blended into a color already specified for pixelP in frame buffer 290 based on depth D and a depth of pixel P in framebuffer 290. In another example, Z-buffering can be used; i.e., if depthD for the fragment is less than the depth of pixel P in frame buffer290, the old color and depth in frame buffer 290 for pixel P can bereplaced, respectively, with color CP and depth D.

In some embodiments, fragment shader 280 can utilize and determinestencil values for pixels in an input fragment. A stencil value can usedto define a rendering area—pixels with zero stencil values can bediscarded and pixels with non-zero stencil values can be made availablefor display Stencil values can be automatically increased or decreasedfor every pixel that fails or passes the depth test. Combining depth andstencil values is used to create effects such as shadowing, drawing ofoutlines, and highlighting of primitive intersections.

In some cases, an input fragment to fragment shader 280 can bediscarded; e.g., the fragment is not visible in the current image. Then,once color, depth, and perhaps stencil values are determined for eachpixel in a non-discarded fragment, the non-discarded fragment can beoutput by fragment shader 280 as a fragment of colored and depthadjustment fragments 282.

Fragment processing 284 can apply tests to each of colored and depthadjustment fragments 282 to determine whether some or all of the pixelsof the fragment are written as pixel(s) 286 to frame buffer 290. Exampletests include a scissors test for only rendering pixels within arectangular stencil window of pixels, an alpha test for testingfragments based on alpha values, stencil and depth testing as discussedimmediately above, a dithering test to determine if colors of pixels areto be dithered, or randomly modified, to increase perceived colorresolution, and logical-operations testing to XOR or otherwise apply alogical operation to pixels in the fragment. After applying these tests,and perhaps other processing, fragment processing 284 can generatepixels 286 that are written into frame buffer 290.

Example Graphics Processing Scenarios

FIG. 3 shows user interface scenario 300 for an example messagingapplication, in accordance with an example embodiment. Scenario 300includes mobile device 210 utilizing user interface (UI) 304 to drawtext, shapes, and images for an example messaging application. Themessaging application utilizes user interface 304 to generateinstructions to draw title bar 310 with background shape (BG) 312 andapplication title 314, buttons 320 on background shape 322, eightmessage summaries shown in FIG. 3 as summaries (“Sums”) 332, 340, 350,352, 354, 356, 358, and 360. Each message summary includes a backgroundshape, a title, a preview, and a graphic. For example, FIG. 3 shows thatsummary 340 of a message includes background shape 342 of a dottedrectangle, title 344 of the message indicating a message sender of“Sender 2”, a preview of the message 346 of “The first few words of thismessage are . . . ”, and a graphic 348 of two adjacent black rectanglesboth pointing to the right.

In scenario 300, user interface 304 includes components of the messagingapplication shown in the first column of Table 1 below, withcorresponding instructions for drawing the components in the secondcolumn of Table 1 below, and drawing VOTs corresponding to theinstructions in the third column of Table 1 below.

TABLE 1 Component Instructions VOTs 1. Title Bar 310 1a. Draw backgroundshape (BG) 312 1a. Shape 1b. Draw title text (Title) 314 1b. Text 2.Buttons 320 2a. Draw BG 322 2a. Shape 2b. Draw image for button (Button)324 2b. Image 2c. Draw Button 326 2c. Image 2d. Draw Button 328 2d.Image 2e. Draw Button 330 2e. Image 3. Summary 332 3a. Draw BG for 3323a. Shape 3b. Draw Title for 332 3b. Text 3c. Draw preview text(Preview) for 3c. Text 332 3d. Draw graphic (Graphic) for 332 3d. Image4. Summary 340 4a. Draw BG 342 4a. Shape 4b. Draw Title 344 4b. Text 4c.Draw Preview 346 4c. Text 4d. Draw Graphic 348 4d. Image 5. Summary 3505a. Draw BG for 350 5a. Shape 5b. Draw Title for 350 5b. Text 5c. DrawPreview for 350 5c. Text 5d. Draw Graphic for 350 5d. Image 6. Summary352 6a. Draw BG for 352 6a. Shape 6b. Draw Title for 352 6b. Text 6c.Draw Preview for 352 6c. Text 6d. Draw Graphic for 352 6d. Image 7.Summary 354 7a. Draw BG for 354 7a. Shape 7b. Draw Title for 354 7b.Text 7c. Draw Preview for 354 7c. Text 7d. Draw Graphic for 354 7d.Image 8. Summary 356 8a. Draw BG for 356 8a. Shape 8b. Draw Title for356 8b. Text 8c. Draw Preview for 356 8c. Text 8d. Draw Graphic for 3568d. Image 9. Summary 358 9a. Draw BG for 358 9a. Shape 9b. Draw Titlefor 358 9b. Text 9c. Draw Preview for 358 9c. Text 9d. Draw Graphic for358 9d. Image 10. Summary 360 10a. Draw BG for 360 10a. Shape 10b. DrawTitle for 360 10b. Text 10c. Draw Preview for 360 10c. Text 10d. DrawGraphic for 360 10d. Image

In scenario 300, the messaging application requests rendering ofcomponents of user interface 304 in the order shown in the first columnof Table 1; that is, first title bar 310 is to be rendered, then buttons320 are to be rendered, and then each of the eight summaries are torendered in an order of the summaries as numbered in FIG. 3—summary 332,then summary 340, and so on to conclude with summary 360.

The second column of Table 1 indicates that, for each component of userinterface 304, between two and five drawing instructions are utilized todraw the component in scenario 300. For example, Table 1 shows that twodrawing instructions are used to draw title bar 310: a drawinginstruction to draw a shape corresponding to background 312 and adrawing instruction to draw text corresponding to title 314. In scenario300, the messaging application causes drawing instructions to begenerated and provided to graphics processing component 720 in the ordershown in the middle column of Table 1. This order includes generatingand providing instruction 1 a first, instruction 1 b second, instruction2 a third, and so on, until instruction 10 d is generated and providedto graphics processing component 720.

The third column of Table 1 shows a drawing VOT associated with eachdrawing instruction. For example, Table 1 shows that the drawinginstruction used to draw background 312 corresponds to a “Shape” VOT 1,the drawing instruction used to draw title 314 corresponds to a “Text”VOT, and the drawing instruction used to draw an image for button 324corresponds to an “Image” VOT.

The instructions in Table 1 can be sorted and/or merged based onsimilarity criteria, such as the VOT associated with each instruction,as shown in Table 2 below. Table 2 has four columns: the instructionsfor scenario 300 to graphics processing component 720 in theoriginally-presented order, the VOTs associated with the instructions tographics processing component 720 in the same order, the instructionsfor scenario 300 to graphics processing component 720 sorted by VOT, andthe instructions for scenario 300 to graphics processing component 720merged by VOT.

TABLE 2 Original Instructions VOTs Sorted Merged 1a. Draw BG 312 1a.Shape Shape Instructions 1a. Draw BG 312 1b. Draw Title 314 1b. Text 1a.Draw BG 312 1b. Draw Title 314 2a. Draw BG 322 2a. Shape 2a. Draw BG 3222a. Draw BG 322 2b. Draw Button 324 2b. Image 3a. Draw BG 332 2b-e. DrawButtons 2c. Draw Button 326 2c. Image 4a. Draw BG 342 324, 326, 328, and330 2d. Draw Button 328 2d. Image 5a. Draw BG 350 3a. Draw BG 332 2e.Draw Button 330 2e. Image 6a. Draw BG 352 3b-c. Draw Title 332 3a. DrawBG 332 3a. Shape 7a. Draw BG 354 and Preview 332 3b. Draw Title 332 3b.Text 8a. Draw BG 356 3d. Draw Graphic 332 3c. Draw Preview 332 3c. Text9a. Draw BG 358 4a. Draw BG 342 3d. Draw Graphic 332 3d. Image 10a. DrawBG 360 4b-c. Draw Title 344 4a. Draw BG 342 4a. Shape Text Instructionsand Preview 346 4b. Draw Title 344 4b. Text 1b. Draw Title 314 4d. DrawGraphic 348 4c. Draw Preview 346 4c. Text 3b. Draw Title 332 5a. Draw BG350 4d. Draw Graphic 348 4d. Image 3c. Draw Preview 332 5b-c. Draw Title350 5a. Draw BG 350 5a. Shape 4b. Draw Title 344 and Preview 350 5b.Draw Title 350 5b. Text 4c. Draw Preview 346 5d. Draw Graphic 350 5c.Draw Preview 350 5c. Text 5b. Draw Title 350 6a. Draw BG 352 5d. DrawGraphic 350 5d. Image 5c. Draw Preview 350 6b-c. Draw Title 352 6a. DrawBG 352 6a. Shape 6b. Draw Title 352 and Preview 352 6b. Draw Title 3526b. Text 6c. Draw Preview 352 6d. Draw Graphic 352 6c. Draw Preview 3526c. Text 7b. Draw Title 354 7a. Draw BG 354 6d. Draw Graphic 352 6d.Image 7c. Draw Preview 354 7b-c. Draw Title 354 7a. Draw BG 354 7a.Shape 8b. Draw Title 356 and Preview 354 7b. Draw Title 354 7b. Text 8c.Draw Preview 356 7d. Draw Graphic 354 7c. Draw Preview 354 7c. Text 9b.Draw Title 358 8a. Draw BG 356 7d. Draw Graphic 354 7d. Image 9c. DrawPreview 358 8b-c. Draw Title 356 8a. Draw BG 356 8a. Shape 10b. DrawTitle 360 and Preview 356 8b. Draw Title 356 8b. Text 10c. Draw Preview360 8d. Draw Graphic 356 8c. Draw Preview 356 8c. Text ImageInstructions 9a. Draw BG 358 8d. Draw Graphic 356 8d. Image 2b. DrawButton 324 9b-c. Draw Title 358 9a. Draw BG 358 9a. Shape 2c. DrawButton 326 and Preview 358 9b. Draw Title 358 9b. Text 2d. Draw Button328 9d. Draw Graphic 358 9c. Draw Preview 358 9c. Text 2e. Draw Button330 10a. Draw BG 360 9d. Draw Graphic 358 9d. Image 3d. Draw Graphic 33210b-c. Draw Title 360 10a. Draw BG 360 10a. Shape 4d. Draw Graphic 348and Preview 360 10b. Draw Title 360 10b. Text 5d. Draw Graphic 350 10d.Draw Graphic 360 10c. Draw Preview 360 10c. Text 6d. Draw Graphic 35210d. Draw Graphic 360 10d. Image 7d. Draw Graphic 354 8d. Draw Graphic356 9d. Draw Graphic 358 10d. Draw Graphic 360

The instructions for scenario 300 to graphics processing component 720and the VOTs associated with the instructions to graphics processingcomponent 720 are copied from Table 1 to increase the readability ofTable 2.

The instructions for scenario 300 to graphics processing component 720are sorted by similarity criteria taking overlap and state into account,such as discussed above in detail in the context of FIGS. 2A and 2B. Inscenario 300, the only overlapping involves drawing text and images ontop of background shapes. Overlap can then be considered by drawing allof the background shapes first, as shown in both the sorted and mergedinstructions above. Also, in scenario 300, no state instructions areprovided by mobile device 210 to graphics processing component 720 fordisplaying components of user interface 304 as shown in FIG. 3.

By sorting instructions by VOT, the number of instructions that operateon the same type of visible object can be increased. In someembodiments, shaders are provided to graphics pipeline 740 based on theVOT; e.g., one or more shaders can be provided to render text/performinstructions associated with a text VOT, while other shader(s) can beprovided to render images/perform instructions associated with imageand/or shape VOTs. In these embodiments, by sorting instructions by VOT,shaders can be provided for a first instruction associated with a givenVOT and then reused by each immediately subsequent instruction that isassociated with the same VOT as the given VOT. As mentioned above, othersimilarity criteria can be used as well to sort and/or mergeinstructions.

The reuse of shaders can decrease time and resources utilized bygraphics pipeline 740 by eliminating the time and resources used toswitch now-reused shaders. Merging instructions can decrease time andresources utilized by graphics pipeline 740 by eliminating the time andresources used to perform per-instruction processing for each of thenow-merged instructions.

The first column of Table 2 indicates that 39 instructions areoriginally provided to graphics processing component 720. In theoriginal order of instructions, the 39 instructions change VOTs 24 timesbetween instructions. For example, instruction 1 a in the first columnof Table 2 has a VOT of “shape” as shown in the corresponding entry ofthe second column of Table 2 and instruction 1 b has a VOT of “text”.Thus, between instruction 1 a and instruction 1 b, the VOT has changedfrom a shape VOT to a text VOT. The third column of Table 2 shows that,after sorting, the VOT only changes three times: the initial VOT of“shape”, then after performing instruction 10 a, the corresponding VOTchanges to “text” for instruction 1 b, and after performing instruction10 c, the corresponding VOT changes to “image” for instruction 2 b.

FIG. 4A shows user interface scenario 400 for the example messagingapplication, in accordance with an example embodiment. In user interfacescenario 400, the original instructions for components of user interface304 are first sorted by mobile device 210 and then provided to graphicsprocessing component 720 for performance and display on display 230. InFIG. 4A each components of user interface 304 is numbered according torendering or drawing order based on the sorted instructions shown inTable 2. That is, background 410 is rendered (drawn) first in accordwith instruction 1 a, then background 412 is rendered, and so on untiltexture 484 is rendered in accord with instruction 10 d.

The fourth column of Table 2 shows that the number of instructions bymerging adjacent instructions based on the similarity criteria of havingthe same VOT is reduced from 39 total instructions as originallypresented to 28 instructions.

In some embodiments, instructions can be both sorted and merged. Forexample, Table 3 below has three columns: the original instructions andsorted instructions shown in Table 2 above, and a “Sorted, the Merged”column showing a result of sorting and the merging the originalinstructions for scenario 300.

TABLE 3 Original Instructions Sorted Sorted, then Merged 1a. Draw BG 312Shapes 1. Draw BG 312, BG 322, 1b. Draw Title 314 1a. Draw BG 312 BG332, BG 342, BG 350, 2a. Draw BG 322 2a. Draw BG 322 BG 352, BG 354, BG356, 2b. Draw Button 324 3a. Draw BG 332 BG 358, and BG 360 2c. DrawButton 326 4a. Draw BG 342 2. Draw Title 314, Title 332, 2d. Draw Button328 5a. Draw BG 350 Preview 332, Title 344, 2e. Draw Button 330 6a. DrawBG 352 Preview 346, Title 350, 3a. Draw BG 332 7a. Draw BG 354 Preview350, Title 352, 3b. Draw Title 332 8a. Draw BG 356 Preview 352, Title354, 3c. Draw Preview 332 9a. Draw BG 358 Preview 354, Title 356, 3d.Draw Graphic 332 10a. Draw BG 360 Preview 356, Title 358, 4a. Draw BG342 Text Preview 358, Title 360, and 4b. Draw Title 344 1b. Draw Title314 Preview 360. 4c. Draw Preview 346 3b. Draw Title 332 3. Draw Button324, Button 4d. Draw Graphic 348 3c. Draw Preview 332 326, Button 328,Button 330, 5a. Draw BG 350 4b. Draw Title 344 Graphic 332, Graphic 348,5b. Draw Title 350 4c. Draw Preview 346 Graphic 350, Graphic 352, 5c.Draw Preview 350 5b. Draw Title 350 Graphic 354, Graphic 356, 5d. DrawGraphic 350 5c. Draw Preview 350 Graphic 358, and Graphic 6a. Draw BG352 6b. Draw Title 352 360. 6b. Draw Title 352 6c. Draw Preview 352 6c.Draw Preview 352 7b. Draw Title 354 6d. Draw Graphic 352 7c. DrawPreview 354 7a. Draw BG 354 8b. Draw Title 356 7b. Draw Title 354 8c.Draw Preview 356 7c. Draw Preview 354 9b. Draw Title 358 7d. DrawGraphic 354 9c. Draw Preview 358 8a. Draw BG 356 10b. Draw Title 360 8b.Draw Title 356 10c. Draw Preview 360 8c. Draw Preview 356 Images 8d.Draw Graphic 356 2b. Draw Button 324 9a. Draw BG 358 2c. Draw Button 3269b. Draw Title 358 2d. Draw Button 328 9c. Draw Preview 358 2e. DrawButton 330 9d. Draw Graphic 358 3d. Draw Graphic 332 10a. Draw BG 3604d. Draw Graphic 348 10b. Draw Title 360 5d. Draw Graphic 350 10c. DrawPreview 360 6d. Draw Graphic 352 10d. Draw Graphic 360 7d. Draw Graphic354 8d. Draw Graphic 356 9d. Draw Graphic 358 10d. Draw Graphic 360

Table 3 shows the three resulting instructions after sorting and thenmerging the original 39 instructions of scenario 300. The threeresulting instructions take overlap and state into account, whilereducing overhead for 24 instruction changes and 39 instructions to theoverhead for three instruction changes and three instructions.

FIG. 4B shows user interface scenario 490 for the example messagingapplication, in accordance with an example embodiment. In user interfacescenario 490, the original instructions for components of user interface304 are first sorted and then merged by mobile device 210. The sortedand merged instructions, shown above in the third column of Table 3, areprovided to graphics processing component 720 for performance anddisplay on display 230.

In FIG. 4B, each component of user interface 304 is numbered andlettered according to rendering order based on the sorted and mergedinstructions. Each of the three sorted and merged instructions is shownwith a separate number: components rendered by instruction 1 of the“Sorted, then Merged”/third column of table 3 are labeled with thenumber 492 followed by a letter indicating the order the component islisted within instruction 1, components rendered by instruction 2 of the“Sorted, then Merged”/third column of table 3 are labeled with thenumber 494 followed by a letter indicating the order the component islisted within instruction 2, and components rendered by instruction 3 ofthe “Sorted, then Merged”/third column of table 3 are labeled with thenumber 496 followed by a letter indicating the order the component islisted within instruction 3. For example, title 494J is performed byinstruction 2 of the third column of Table 3 and is the 10^(th)component listed within instruction 2 (corresponding to the ordinalposition of letter J within the alphabet) or “Title [for originalsummary] 354 [as numbered in FIG. 3]”.

The reordering and/or merging of graphics processing componentinstructions can be performed without changing the software in themessaging application or changing appearance or performance of userinterface 304. As shown in the examples above, 39 instructions used todraw user interface 304 can be reduced to as few as 3 instructions.Thus, the performance of user interface 304 can be improved, in somecases substantially improved, without changing the software of userinterface 304.

FIG. 5 shows user interface scenario 500 for an example drawingapplication 510, in accordance with an example embodiment. Userinterface 504 of drawing application 510 includes a title bar background512 of a rectangle with a dotted background, title bar text 514 of “SomeDrawing Application . . . ”, and a drawing region. The drawing regionshows circles 516, 518, 502, 522, 524, 526, 528, 532, 534, 536, 538,540, 542, and 544 of having various background textures, text 530 of“Our Latest Circular!”, text 546 of “Lowest Prices Ever!”, and text 548of “C & R's Baubles—Bubbles and Baubles of All Kinds . . . ”.

As shown in FIG. 5, text 514 overlaps title bar background 512, circle518 overlaps circle 516, circle 520 overlaps circle 518, circle 524overlaps circle 522, circle 526 overlaps circle 524, circle 528 overlapscircles 518, 520, 522, and 524, and text 530 overlaps circle 528. FIG. 5also shows that circle 534 overlaps circle 532, circle 536 overlapscircle 534, circle 540 overlaps circle 538, circle 542 overlaps circle540, circle 544 overlaps circles 534, 536, 538, 540, and 542, and text546 overlaps circle 544.

TABLE 4 Instructions VOTs Sorted Sorted, then Merged 1. Draw BG 512 1.Shape 1. Draw BG 512 1. Draw BG 512, Circle 516, 2. Draw Title 514 2.Text 2. Draw Circle 516 Circle 522, Circle 532, Circle 538 3. DrawCircle 516 3. Shape 3. Draw Circle 532 2. Draw Title 514, Text 548 4.Draw Circle 518 4. Shape 4. Draw Title 514 3. Draw Circle 518, Circle524, 5. Draw Circle 520 5. Shape 5. Draw Text 548 Circle 534, Circle 5406. Draw Circle 522 6. Shape 6. Draw Circle 518 4. Draw Circle 520,Circle 526, 7. Draw Circle 524 7. Shape 7. Draw Circle 534 Circle 536,Circle 542 8. Draw Circle 526 8. Shape 8. Draw Circle 520 5. Draw Circle528, Circle 544 9. Draw Circle 528 9. Shape 9. Draw Circle 536 6. DrawText 530, Text 546 10. Draw Text 530 10. Text 10. Draw Circle 522 11.Draw Circle 532 11. Shape 11. Draw Circle 538 12. Draw Circle 534 12.Shape 12. Draw Circle 524 13. Draw Circle 536 13. Shape 13. Draw Circle540 14. Draw Circle 538 14. Shape 14. Draw Circle 526 15. Draw Circle540 15. Shape 15. Draw Circle 542 16. Draw Circle 542 16. Shape 16. DrawCircle 528 17. Draw Circle 544 17. Shape 17. Draw Circle 544 18. DrawText 546 18. Text 18. Draw Text 530 19. Draw Text 548 19. Text 19. DrawText 546

Table 4 includes four columns: the first column shows example commandsto graphics processing unit 720 for displaying the image presented byuser interface 504 of drawing application 510, the second column showseach VOT associated with the corresponding instruction in the firstcolumn, the third column shows the instructions of the first columnafter being sorted by the similarity criteria of VOT while taking stateand overlap into account, and the fourth column shows the instructionsof the first column after being sorted and merged while taking state andoverlap into account.

During sorting, a draw circle 518 instruction can be determined tooverlap a draw circle 516 instruction. Then, to preserve correctness ofthe image presented by user interface 504, the draw circle 518instruction should take place after the draw circle 518 instruction.Further, as in some embodiments, the order of merged instructions can beindeterminate, the draw circle 516 instruction and the draw circle 518instruction should remain separate; i.e., not merged. Similarly, theother overlapping circles and text noted above indicates that fewerinstructions can be merged due to overlap considerations.

The first column of Table 4 shows 19 example instructions to graphicsprocessing component 720 for drawing components of user interface 504shown in FIG. 5. The 19 commands include five VOT changes: a change froma shape VOT to a text VOT between instructions 1 and 2, a change from atext VOT to a shape VOT between instructions 2 and 3, a change from ashape VOT to a text VOT between instructions 9 and 10, a change from atext VOT to a shape VOT between instructions 10 and 11, and a changefrom a shape VOT to a text VOT between instructions 17 and 18. Aftersorting, only three VOT changes occur: a change from a shape VOT to atext VOT between sorted instructions 3 and 4, a change from a text VOTto a shape VOT between sorted instructions 5 and 6, and a change from ashape VOT to a text VOT between instructions 17 and 18.

After sorting and merging the example instructions to graphicsprocessing component 720 for drawing components of user interface 504,the fourth column shows a total of six resulting instructions with threechanges in VOT, leading to a net reduction of thirteen instructions andtwo changes in VOT. Therefore, even when provided with instructions foran image with a relatively large number of overlapping components, suchas the image of user interface 504 shown in FIG. 5, 13 of the original19 instructions and two of the original five changes in VOT can beeliminated, saving about one-half of the overhead required forinstructions and changes in VOT. By sorting and merging of GUI commands,overhead can be reduced by almost one-half even for images wheresignificant overhead reduction may not be expected due to features ofthe image being rendered.

Example Data Network

FIG. 6 shows server devices 608, 610 configured to communicate, vianetwork 606, with programmable devices 604 a, 604 b, and 604 c. Network606 may correspond to a LAN, a wide area network (WAN), a corporateintranet, the public Internet, or any other type of network configuredto provide a communications path between networked computing devices.The network 606 may also correspond to a combination of one or moreLANs, WANs, corporate intranets, and/or the public Internet.

Although FIG. 6 only shows three programmable devices, distributedapplication architectures may serve tens, hundreds, or thousands ofprogrammable devices. Moreover, programmable devices 604 a, 604 b, and604 c (or any additional programmable devices) may be any sort ofcomputing device, such as an ordinary laptop computer, desktop computer,network terminal, wireless communication device (e.g., a cell phone orsmart phone), and so on. In some embodiments, programmable devices 604a, 604 b, and 604 c may be dedicated to the design and use of softwareapplications. In other embodiments, programmable devices 604 a, 604 b,and 604 c may be general purpose computers that are configured toperform a number of tasks and need not be dedicated to softwaredevelopment tools.

Server devices 608, 610 can be configured to perform one or moreservices, as requested by programmable devices 604 a, 604 b, and/or 604c. For example, server device 608 and/or 610 can provide content toprogrammable devices 604 a-604 c. The content can include, but is notlimited to, web pages, hypertext, scripts, binary data such as compiledsoftware, images, audio, and/or video.

The content can include compressed and/or uncompressed content. Thecontent can be encrypted and/or unencrypted. Other types of content arepossible as well.

As another example, server device 608 and/or 610 can provideprogrammable devices 604 a-604 c with access to software for database,search, computation, graphical, audio, video, World Wide Web/Internetutilization, and/or other functions. Many other examples of serverdevices are possible as well.

Computing Device Architecture

FIG. 7A is a block diagram of a computing device (e.g., system) inaccordance with an example embodiment. In particular, computing device700 shown in FIG. 7A can be configured to perform one or more functionsof mobile device 210, server devices 608, 610, network 606, and/or oneor more programmable devices 604 a, 604 b, and 604 c. Computing device700 may include a user interface module 701, a network-communicationinterface module 702, one or more processors 703, data storage 704, andgraphics processing component 720, all of which may be linked togethervia a system bus, network, or other connection mechanism 705.

User interface module 701 can be operable to send data to and/or receivedata from external user input/output devices. For example, userinterface module 701 can be configured to send and/or receive data toand/or from user input devices such as a keyboard, a keypad, a touchscreen, a computer mouse, a track ball, a joystick, a camera, a voicerecognition module, and/or other similar devices. User interface module701 can also be configured to provide output to user display devices,such as one or more cathode ray tubes (CRT), liquid crystal displays(LCD), light emitting diodes (LEDs), displays using digital lightprocessing (DLP) technology, printers, light bulbs, and/or other similardevices, either now known or later developed. For example, in someembodiments, user interface module 701 can include one or more canvasesfor providing and/or displaying visible output, such as but not limitedto, canvas 230, display 230, 230 a, and/or frame buffer 290. Userinterface module 701 can also be configured to generate audibleoutput(s), such as a speaker, speaker jack, audio output port, audiooutput device, earphones, and/or other similar devices.

Network-communications interface module 702 can include one or morewireless interfaces 707 and/or one or more wireline interfaces 708 thatare configurable to communicate via a network, such as network 606 shownin FIG. 6. Wireless interfaces 707 can include one or more wirelesstransmitters, receivers, and/or transceivers, such as a Bluetooth®transceiver, a Zigbee® transceiver, a Wi-Fi® transceiver, a WiMAX™transceiver, and/or other similar type of wireless transceiverconfigurable to communicate via a wireless network. Wireline interfaces708 can include one or more wireline transmitters, receivers, and/ortransceivers, such as an Ethernet transceiver, a Universal Serial Bus(USB) transceiver, or similar transceiver configurable to communicatevia a twisted pair wire, a coaxial cable, a fiber-optic link, or asimilar physical connection to a wireline network.

In some embodiments, network communications interface module 702 can beconfigured to provide reliable, secured, and/or authenticatedcommunications. For each communication described herein, information forensuring reliable communications (i.e., guaranteed message delivery) canbe provided, perhaps as part of a message header and/or footer (e.g.,packet/message sequencing information, encapsulation header(s) and/orfooter(s), size/time information, and transmission verificationinformation such as CRC and/or parity check values). Communications canbe made secure (e.g., be encoded or encrypted) and/or decrypted/decodedusing one or more cryptographic protocols and/or algorithms, such as,but not limited to, DES, AES, RSA, Diffie-Hellman, and/or DSA. Othercryptographic protocols and/or algorithms can be used as well or inaddition to those listed herein to secure (and then decrypt/decode)communications.

Processors 703 can include one or more general purpose processors and/orone or more special purpose processors (e.g., digital signal processors,application specific integrated circuits, etc.). Processors 703 can beconfigured to execute computer-readable program instructions 706 a thatare contained in the data storage 704 and/or other instructions asdescribed herein.

Data storage 704 can include one or more computer-readable storage mediathat can be read and/or accessed by at least one of processors 703. Theone or more computer-readable storage media can include volatile and/ornon-volatile storage components, such as optical, magnetic, organic orother memory or disc storage, which can be integrated in whole or inpart with at least one of processors 703. In some embodiments, datastorage 704 can be implemented using a single physical device (e.g., oneoptical, magnetic, organic or other memory or disc storage unit), whilein other embodiments, data storage 704 can be implemented using two ormore physical devices. The one or more computer-readable storage mediacan be, or can include, one or more non-transitory computer-readablestorage media.

Data storage 704 can include computer-readable program instructions 706and perhaps additional data, such as but not limited to data used by oneor more processes and/or threads of a software application. In someembodiments, data storage 704 can additionally include storage requiredto perform at least part of the herein-described methods and techniquesand/or at least part of the functionality of the herein-describeddevices and networks.

Graphics processing component 720 can be, and or include, hardware,firmware, and/or software configured to perform some or all of the tasksused in rendering imagery, perhaps for display using a user displaydevice of user interface module 701. For example, graphics processingcomponent 720 can include, but is not limited to, one or more graphicsprocessing units (GPUs), graphics co-processors, graphics pipelines,frame buffers, graphics libraries, graphics software, computer-readablememories (including non-transitory computer-readable memories) partiallyor entirely dedicated for graphics operations, and rendering engines.For example, in some embodiments, graphics processing component 720 caninclude graphics pipeline 240 and, perhaps frame buffer 290. In someembodiments, part or all of the functionality of graphics processingcomponent 720 can be performed by user interface module 701, processors703, data storage 704, and/or instructions 706.

Cloud-Based Servers

FIG. 7B depicts a network 606 of computing clusters 709 a, 709 b, 709 carranged as a cloud-based server system in accordance with an exampleembodiment. Server devices 608 and/or 610 can be cloud-based devicesthat store program logic and/or data of cloud-based applications and/orservices. In some embodiments, server devices 608 and/or 610 can be asingle computing device residing in a single computing center. In otherembodiments, server device 608 and/or 610 can include multiple computingdevices in a single computing center, or even multiple computing deviceslocated in multiple computing centers located in diverse geographiclocations. For example, FIG. 6 depicts each of server devices 608 and610 residing in different physical locations.

In some embodiments, data and services at server devices 608 and/or 610can be encoded as computer readable information stored innon-transitory, tangible computer readable media (or computer readablestorage media) and accessible by programmable devices 604 a, 604 b, and604 c, and/or other computing devices. In some embodiments, data atserver device 608 and/or 610 can be stored on a single disk drive orother tangible storage media, or can be implemented on multiple diskdrives or other tangible storage media located at one or more diversegeographic locations.

FIG. 7B depicts a cloud-based server system in accordance with anexample embodiment. In FIG. 7B, the functions of server device 608and/or 610 can be distributed among three computing clusters 709 a, 709b, and 708 c. Computing cluster 709 a can include one or more computingdevices 700 a, cluster storage arrays 710 a, and cluster routers 711 aconnected by a local cluster network 712 a. Similarly, computing cluster709 b can include one or more computing devices 700 b, cluster storagearrays 710 b, and cluster routers 711 b connected by a local clusternetwork 712 b. Likewise, computing cluster 709 c can include one or morecomputing devices 700 c, cluster storage arrays 710 c, and clusterrouters 711 c connected by a local cluster network 712 c.

In some embodiments, each of the computing clusters 709 a, 709 b, and709 c can have an equal number of computing devices, an equal number ofcluster storage arrays, and an equal number of cluster routers. In otherembodiments, however, each computing cluster can have different numbersof computing devices, different numbers of cluster storage arrays, anddifferent numbers of cluster routers. The number of computing devices,cluster storage arrays, and cluster routers in each computing clustercan depend on the computing task or tasks assigned to each computingcluster.

In computing cluster 709 a, for example, computing devices 700 a can beconfigured to perform various computing tasks of server 608. In oneembodiment, the various functionalities of server 608 can be distributedamong one or more computing devices 700 a, 700 b, and 700 c. Computingdevices 700 b and 700 c in computing clusters 709 b and 709 c can beconfigured similarly to computing devices 700 a in computing cluster 709a. On the other hand, in some embodiments, computing devices 700 a, 700b, and 700 c can be configured to perform different functions.

In some embodiments, computing tasks and stored data associated withserver devices 608 and/or 610 can be distributed across computingdevices 700 a, 700 b, and 700 c based at least in part on the processingrequirements of server devices 608 and/or 610, the processingcapabilities of computing devices 700 a, 700 b, and 700 c, the latencyof the network links between the computing devices in each computingcluster and between the computing clusters themselves, and/or otherfactors that can contribute to the cost, speed, fault-tolerance,resiliency, efficiency, and/or other design goals of the overall systemarchitecture.

The cluster storage arrays 710 a, 710 b, and 710 c of the computingclusters 709 a, 709 b, and 709 c can be data storage arrays that includedisk array controllers configured to manage read and write access togroups of hard disk drives. The disk array controllers, alone or inconjunction with their respective computing devices, can also beconfigured to manage backup or redundant copies of the data stored inthe cluster storage arrays to protect against disk drive or othercluster storage array failures and/or network failures that prevent oneor more computing devices from accessing one or more cluster storagearrays.

Similar to the manner in which the functions of server devices 608and/or 610 can be distributed across computing devices 700 a, 700 b, and700 c of computing clusters 709 a, 709 b, and 709 c, various activeportions and/or backup portions of these components can be distributedacross cluster storage arrays 710 a, 710 b, and 710 c. For example, somecluster storage arrays can be configured to store the data of serverdevice 608, while other cluster storage arrays can store data of serverdevice 610. Additionally, some cluster storage arrays can be configuredto store backup versions of data stored in other cluster storage arrays.

The cluster routers 711 a, 711 b, and 711 c in computing clusters 709 a,709 b, and 709 c can include networking equipment configured to provideinternal and external communications for the computing clusters. Forexample, the cluster routers 711 a in computing cluster 709 a caninclude one or more internet switching and routing devices configured toprovide (i) local area network communications between the computingdevices 700 a and the cluster storage arrays 701 a via the local clusternetwork 712 a, and (ii) wide area network communications between thecomputing cluster 709 a and the computing clusters 709 b and 709 c viathe wide area network connection 713 a to network 606. Cluster routers711 b and 711 c can include network equipment similar to the clusterrouters 711 a, and cluster routers 711 b and 711 c can perform similarnetworking functions for computing clusters 709 b and 709 b that clusterrouters 711 a perform for computing cluster 709 a.

In some embodiments, the configuration of the cluster routers 711 a, 711b, and 711 c can be based at least in part on the data communicationrequirements of the computing devices and cluster storage arrays, thedata communications capabilities of the network equipment in the clusterrouters 711 a, 711 b, and 711 c, the latency and throughput of localnetworks 712 a, 712 b, 712 c, the latency, throughput, and cost of widearea network links 713 a, 713 b, and 713 c, and/or other factors thatcan contribute to the cost, speed, fault-tolerance, resiliency,efficiency and/or other design goals of the moderation systemarchitecture.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The illustrativeembodiments described in the detailed description, figures, and claimsare not meant to be limiting. Other embodiments can be utilized, andother changes can be made, without departing from the spirit or scope ofthe subject matter presented herein. It will be readily understood thatthe aspects of the present disclosure, as generally described herein,and illustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are explicitly contemplated herein.

With respect to any or all of the ladder diagrams, scenarios, and flowcharts in the figures and as discussed herein, each block and/orcommunication may represent a processing of information and/or atransmission of information in accordance with example embodiments.Alternative embodiments are included within the scope of these exampleembodiments. In these alternative embodiments, for example, functionsdescribed as blocks, transmissions, communications, requests, responses,and/or messages may be executed out of order from that shown ordiscussed, including substantially concurrent or in reverse order,depending on the functionality involved. Further, more or fewer blocksand/or functions may be used with any of the ladder diagrams, scenarios,and flow charts discussed herein, and these ladder diagrams, scenarios,and flow charts may be combined with one another, in part or in whole.

A block that represents a processing of information may correspond tocircuitry that can be configured to perform the specific logicalfunctions of a herein-described method or technique. Alternatively oradditionally, a block that represents a processing of information maycorrespond to a module, a segment, or a portion of program code(including related data). The program code may include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data may be stored on any type of computer readable medium suchas a storage device including a disk or hard drive or other storagemedium.

The computer readable medium may also include non-transitory computerreadable media such as non-transitory computer-readable media thatstores data for short periods of time like register memory, processorcache, and random access memory (RAM). The computer readable media mayalso include non-transitory computer readable media that stores programcode and/or data for longer periods of time, such as secondary orpersistent long term storage, like read only memory (ROM), optical ormagnetic disks, compact-disc read only memory (CD-ROM), for example. Thecomputer readable media may also be any other volatile or non-volatilestorage systems. A computer readable medium may be considered a computerreadable storage medium, for example, or a tangible storage device.

Moreover, a block that represents one or more information transmissionsmay correspond to information transmissions between software and/orhardware modules in the same physical device. However, other informationtransmissions may be between software modules and/or hardware modules indifferent physical devices.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1-20. (canceled)
 21. A method, comprising: receiving, at a computingdevice, a plurality of instructions for a graphics processing componentof the computing device, wherein the graphics processing component isassociated with a plurality of shaders, wherein the plurality ofinstructions comprise a first instruction associated with a first groupof shaders selected from the plurality of shaders and a secondinstruction associated with a second group of shaders selected from theplurality of shaders; determining a sorted plurality of instructions byat least sorting the plurality of instructions based on the first andsecond groups of shaders; determining one or more merged instructions byat least merging the sorted plurality of instructions; and providing theone or more merged instructions to the graphics processing component,wherein the graphics processing component performs the one or moremerged instructions.
 22. The method of claim 21, wherein eachinstruction of the plurality of instructions are associated with atleast a portion of a canvas, and wherein sorting the plurality ofinstructions comprises: determining a first portion of the canvasassociated with the first instruction; determining an insertion positionin an instruction buffer for the first instruction by at least:determining a second instruction in the instruction buffer; determininga second portion of the canvas associated with the second instruction inthe instruction buffer; and if the second portion overlaps the firstportion, then determining the insertion position based on anoverlapping-instruction position, wherein the overlapping-instructionposition is based on a position in the instruction buffer of the secondinstruction in the instruction buffer.
 23. The method of claim 21,wherein sorting the plurality of instructions comprises: determiningwhether the first instruction is similar to the second instruction basedon the first and second groups of shaders; and after determining thatthe first instruction is similar to the second instruction, determiningan insertion position in an instruction buffer based on a secondposition, wherein the second position is based on the position in theinstruction buffer of the second instruction in the instruction buffer.24. The method of claim 23, wherein determining whether the firstinstruction is similar to the second instruction based on the first andsecond groups of shaders comprises determining that the firstinstruction is similar to the second instruction when the first group ofthe one or more shaders is the same as the second group of shaders. 25.The method of claim 23, wherein the instruction buffer has a beginningposition that begins the instruction buffer and an ending position endsthe instruction buffer, wherein the instruction buffer stores at leastone instruction, and wherein determining the insertion positioncomprises: setting a search position to an initial search position thatis based on the ending position; determining a third instruction at thesearch position in the instruction buffer; determining a third portionof the canvas associated with the third instruction at the searchposition of the instruction buffer; determining whether the thirdportion overlaps the first portion; after determining that the thirdportion overlaps the first portion, determining that theoverlapping-instruction position is the search position; and otherwise:determining that the third portion does not overlap the first portion,determining whether the third instruction at the search position of theinstruction buffer is similar to the first instruction, afterdetermining that the third instruction at the search position of theinstruction buffer is identified as being similar to the firstinstruction, then determining that the second position is the searchposition, and otherwise: determining that the third portion does notoverlap the first portion and that the third instruction at the searchposition of the instruction buffer is not similar to the firstinstruction, and decrementing the search position.
 26. The method ofclaim 25, further comprising: after decrementing the search position,determining whether the search position is before the beginningposition; and after determining that the search position is before thebeginning position: determining a position of a most-similarinstruction, wherein the most-similar instruction is an instruction thatis closest in similarity to an instruction that has been searched in theinstruction buffer, and determining that the insertion position is basedon the position of the most-similar instruction.
 27. The method of claim21, wherein merging the sorted plurality of instructions comprises:determining a first plurality of consecutive instructions of the sortedplurality of instructions, wherein each instruction of the firstplurality of consecutive instructions is associated with a third groupof shaders; and determining a first merged instruction of the one ormore merged instructions by at least merging the first plurality ofconsecutive instructions into the first merged instructions.
 28. Themethod of claim 21, wherein the sorted plurality of instructions arestored in an instruction buffer, and wherein merging the sortedplurality of instructions comprises: determining a first state of thegraphics processing component, wherein the first state is represented bya first matrix; associating the first state with a first sortedinstruction of the sorted plurality of instructions; searching aninstruction buffer to identify a mergeable instruction for the firstsorted instruction, wherein the mergeable instruction is stored at amergeable location in the instruction buffer, and wherein the mergeableinstruction is associated with a second state of the graphics processingcomponent, and wherein the second state is represented by a secondmatrix; determining whether the mergeable instruction is to be mergedwith the first sorted instruction based on the first and second states;and after determining that the mergeable instruction is to be mergedwith the first sorted instruction, merging the first sorted instructionwith the mergeable instruction.
 29. The method of claim 28, wherein atleast the first matrix aggregates information about transformations forthe graphics processing component.
 30. The method of claim 28, whereineach instruction of the sorted plurality of instructions is associatedwith a canvas, wherein the first sorted instruction is associated with afirst portion of the canvas, and wherein determining whether themergeable instruction is to be merged with the first sorted instructionbased on the first and second states comprises: searching one or moreinstructions in the instruction buffer for overlap in the canvas withthe first portion, wherein the one or more instructions in theinstruction buffer comprise the mergeable instruction; and determiningwhether the one or more instructions in the instruction buffer do notoverlap the first portion of the canvas and whether the first state doesnot differ from the second state.
 31. The method of claim 30, furthercomprising: after determining the one or more instructions in theinstruction buffer do not overlap the first portion of the canvas andthat first state does not differ from the second state, merging thefirst sorted instruction with the mergeable instruction.
 32. The methodof claim 28, wherein the first sorted instruction is associated with afirst visible object type, and wherein searching the instruction bufferto identify the mergeable instruction: forming a first merging key forthe first sorted instruction, wherein the merging key comprises thefirst visible object type for the instruction; performing a hash lookupin a hash table associated with the instruction buffer utilizing thefirst merging key; and in response to finding a match in the hash tablefor the first merging key, determining that the mergeable instruction isan instruction in the hash table associated with the first merging key.33. The method of claim 32, wherein the first visible object type is oneor more of: a point visible object type, a line visible object type, ashape visible object type, an image visible object type, and a textvisible object type.
 34. The method of claim 21, wherein merging theinstruction and the mergeable instruction comprises: generating a mergedinstruction by merging the instruction and the mergeable instruction;and replacing the mergeable instruction in the instruction buffer withthe merged instruction.
 35. The method of claim 21, wherein the graphicsprocessing component performs the instruction using a renderingpipeline, and wherein the rendering pipeline comprises the one or moreshaders.
 36. The method of claim 21, wherein providing the one or moremerged instructions to the graphics processing component comprisesproviding information about one or more shaders for use in the renderingpipeline to perform the one or more merged instructions.
 37. The methodof claim 21, wherein the plurality of shaders include one or more of avertex shader, a geometry shader, and a fragment shader.
 38. A computingdevice, comprising: a graphics processing component; a processor; andnon-transitory computer-readable memory having one or more executableinstructions stored thereon that, when executed by the processor, causethe computing device to perform functions comprising: receiving aplurality of instructions for the graphics processing component, whereinthe graphics processing component is associated with a plurality ofshaders, wherein the plurality of instructions comprise a firstinstruction associated with a first group of shaders selected from theplurality of shaders and a second instruction associated with a secondgroup of shaders selected from the plurality of shaders; determining asorted plurality of instructions by at least sorting the plurality ofinstructions based on the first and second groups of shaders;determining one or more merged instructions by at least merging thesorted plurality of instructions; and providing the one or more mergedinstructions to the graphics processing component, wherein the graphicsprocessing component performs the one or more merged instructions. 39.The computing device of claim 38, wherein sorting the plurality ofinstructions comprises: determining whether the first instruction issimilar to the second instruction by at least determining that the firstgroup of the one or more shaders is the same as the second group ofshaders; after determining that the first instruction is similar to thesecond instruction, determining an insertion position in an instructionbuffer based on a second position, wherein the second position is basedon the position in the instruction buffer of the second instruction inthe instruction buffer.
 40. The computing device of claim 38, whereinthe sorted plurality of instructions are stored in an instructionbuffer, and wherein merging the sorted plurality of instructionscomprises: determining a first state of the graphics processingcomponent, wherein the first state is represented by a first matrix;associating the first state with a first sorted instruction of thesorted plurality of instructions; searching an instruction buffer toidentify a mergeable instruction for the first sorted instruction,wherein the mergeable instruction is stored at a mergeable location inthe instruction buffer, and wherein the mergeable instruction isassociated with a second state of the graphics processing component, andwherein the second state is represented by a second matrix; determiningwhether the mergeable instruction is to be merged with the first sortedinstruction based on the first and second states; and after determiningthat the mergeable instruction is to be merged with the first sortedinstruction, merging the first sorted instruction with the mergeableinstruction.
 41. An article of manufacture including a non-transitorycomputer-readable memory having one or more executable instructionsstored thereon that, when executed by a processor of a computing device,cause the computing device to perform functions comprising: receiving aplurality of instructions for a graphics processing component of thecomputing device, wherein the graphics processing component isassociated with a plurality of shaders, wherein the plurality ofinstructions comprise a first instruction associated with a first groupof shaders selected from the plurality of shaders and a secondinstruction associated with a second group of shaders selected from theplurality of shaders; determining a sorted plurality of instructions byat least sorting the plurality of instructions based on the first andsecond groups of shaders; determining one or more merged instructions byat least merging the sorted plurality of instructions; and providing theone or more merged instructions to the graphics processing component,wherein the graphics processing component performs the one or moremerged instructions.